This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new c683db636b1b CAMEL-27064: camel-google-storage, camel-minio,
camel-alibaba-mns - the consumers delete or move the object or message they
consumed, whatever the route did with its headers (#30227)
c683db636b1b is described below
commit c683db636b1b4ce1d481e131aa091663b7f8e393
Author: Federico Mariani <[email protected]>
AuthorDate: Mon Nov 16 22:54:16 2026 +0100
CAMEL-27064: camel-google-storage, camel-minio, camel-alibaba-mns - the
consumers delete or move the object or message they consumed, whatever the
route did with its headers (#30227)
Co-authored-by: Claude <[email protected]>
---
.../camel/catalog/docs/alibaba-mns-component.adoc | 4 +
.../catalog/docs/google-storage-component.adoc | 4 +
.../apache/camel/catalog/docs/minio-component.adoc | 3 +
.../src/main/docs/alibaba-mns-component.adoc | 4 +
.../camel/component/alibaba/mns/MNSConsumer.java | 24 ++-
.../alibaba/mns/MNSConsumerDeleteMessageTest.java | 151 +++++++++++++++++
.../src/main/docs/google-storage-component.adoc | 4 +
.../google/storage/GoogleCloudStorageConsumer.java | 28 ++--
...loudStorageConsumerDeleteMoveAfterReadTest.java | 180 +++++++++++++++++++++
.../camel-minio/src/main/docs/minio-component.adoc | 3 +
.../camel/component/minio/MinioConsumer.java | 26 ++-
.../apache/camel/component/minio/FakeS3Server.java | 77 ++++++++-
.../MinioConsumerDeleteMoveAfterReadTest.java | 178 ++++++++++++++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 17 ++
14 files changed, 681 insertions(+), 22 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/alibaba-mns-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/alibaba-mns-component.adoc
index 0c3358680eac..4123227deb08 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/alibaba-mns-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/alibaba-mns-component.adoc
@@ -182,6 +182,10 @@ YAML::
----
====
+With `deleteAfterRead` (the default), the consumer deletes the message once
its exchange completes, with the receipt
+handle it received the message with, whatever the route does with the
`CamelAlibabaMnsReceiptHandle` header. If the
+exchange fails, the message is not deleted, and it is received again once it
is visible again.
+
=== Exchange properties evaluated by the producer
[width="100%",cols="10%,10%,80%",options="header",]
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-storage-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-storage-component.adoc
index 4678ee29f9a0..338b384f679e 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-storage-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-storage-component.adoc
@@ -657,4 +657,8 @@ YAML::
In this case, the objects consumed will be moved to myCamelProcessedBucket
bucket and deleted from the original one (because of deleteAfterRead).
+The consumer deletes or moves the object it consumed, whatever the route does
with the `CamelGoogleCloudStorageObjectName`
+and `CamelGoogleCloudStorageBucketName` headers. A route can set the object
name header to write a copy of the object
+under another name.
+
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/minio-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/minio-component.adoc
index 97527f3c21a9..6e5dc2d0c42f 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/minio-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/minio-component.adoc
@@ -692,6 +692,9 @@ YAML::
In this case, the objects consumed will be moved to `myothercamelbucket`
bucket and deleted from the original one (because of `deleteAfterRead` set to
true as default).
+The consumer deletes or moves the object it consumed, whatever the route does
with the `CamelMinioObjectName` and
+`CamelMinioBucketName` headers. A route can set the object name header to
write a copy of the object under another name.
+
=== Using a POJO as body
Sometimes build a Minio Request can be complex because of multiple options.
diff --git
a/components/camel-alibaba/camel-alibaba-mns/src/main/docs/alibaba-mns-component.adoc
b/components/camel-alibaba/camel-alibaba-mns/src/main/docs/alibaba-mns-component.adoc
index 0c3358680eac..4123227deb08 100644
---
a/components/camel-alibaba/camel-alibaba-mns/src/main/docs/alibaba-mns-component.adoc
+++
b/components/camel-alibaba/camel-alibaba-mns/src/main/docs/alibaba-mns-component.adoc
@@ -182,6 +182,10 @@ YAML::
----
====
+With `deleteAfterRead` (the default), the consumer deletes the message once
its exchange completes, with the receipt
+handle it received the message with, whatever the route does with the
`CamelAlibabaMnsReceiptHandle` header. If the
+exchange fails, the message is not deleted, and it is received again once it
is visible again.
+
=== Exchange properties evaluated by the producer
[width="100%",cols="10%,10%,80%",options="header",]
diff --git
a/components/camel-alibaba/camel-alibaba-mns/src/main/java/org/apache/camel/component/alibaba/mns/MNSConsumer.java
b/components/camel-alibaba/camel-alibaba-mns/src/main/java/org/apache/camel/component/alibaba/mns/MNSConsumer.java
index 0f22de39b2a1..e34f4174591c 100644
---
a/components/camel-alibaba/camel-alibaba-mns/src/main/java/org/apache/camel/component/alibaba/mns/MNSConsumer.java
+++
b/components/camel-alibaba/camel-alibaba-mns/src/main/java/org/apache/camel/component/alibaba/mns/MNSConsumer.java
@@ -123,10 +123,13 @@ public class MNSConsumer extends
ScheduledBatchPollingConsumer {
pendingExchanges = total - index - 1;
+ // the receipt handle to delete the message with once the exchange
is done. It is taken now, as the route
+ // may remove the headers or replace the message before it ends
+ final String receiptHandle =
exchange.getIn().getHeader(MNSHeaders.RECEIPT_HANDLE, String.class);
exchange.getExchangeExtension().addOnCompletion(new
Synchronization() {
@Override
public void onComplete(Exchange exchange) {
- processCommit(exchange);
+ processCommit(exchange, receiptHandle);
}
@Override
@@ -147,12 +150,29 @@ public class MNSConsumer extends
ScheduledBatchPollingConsumer {
return total;
}
+ /**
+ * Strategy to delete the message after being processed, with the receipt
handle of the
+ * {@link MNSHeaders#RECEIPT_HANDLE} header.
+ *
+ * @param exchange the exchange
+ * @deprecated use {@link #processCommit(Exchange, String)},
which the consumer calls instead
+ */
+ @Deprecated(since = "4.24.0")
protected void processCommit(Exchange exchange) {
+ processCommit(exchange,
exchange.getIn().getHeader(MNSHeaders.RECEIPT_HANDLE, String.class));
+ }
+
+ /**
+ * Strategy to delete the message after being processed.
+ *
+ * @param exchange the exchange
+ * @param receiptHandle the receipt handle the message of the exchange was
received with
+ */
+ protected void processCommit(Exchange exchange, String receiptHandle) {
if (!getEndpoint().isDeleteAfterRead()) {
return;
}
try {
- String receiptHandle =
exchange.getIn().getHeader(MNSHeaders.RECEIPT_HANDLE, String.class);
if (ObjectHelper.isEmpty(receiptHandle)) {
return;
}
diff --git
a/components/camel-alibaba/camel-alibaba-mns/src/test/java/org/apache/camel/component/alibaba/mns/MNSConsumerDeleteMessageTest.java
b/components/camel-alibaba/camel-alibaba-mns/src/test/java/org/apache/camel/component/alibaba/mns/MNSConsumerDeleteMessageTest.java
new file mode 100644
index 000000000000..8a18a507e1dc
--- /dev/null
+++
b/components/camel-alibaba/camel-alibaba-mns/src/test/java/org/apache/camel/component/alibaba/mns/MNSConsumerDeleteMessageTest.java
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.alibaba.mns;
+
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.function.Consumer;
+
+import com.aliyun.mns.client.CloudQueue;
+import com.aliyun.mns.client.MNSClient;
+import com.aliyun.mns.model.Message;
+import org.apache.camel.CamelContext;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.alibaba.mns.constants.MNSHeaders;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.model.RouteDefinition;
+import org.apache.camel.support.DefaultMessage;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * With deleteAfterRead the consumer deletes the message it received once the
exchange is done, with the receipt handle
+ * it received the message with, whatever the route did with the
CamelAlibabaMnsReceiptHandle header in the meantime.
+ * <p>
+ * The scheduler of the consumer is not started: the tests call {@code poll()}
themselves. The routes are synchronous,
+ * so the delete is done when {@code poll()} returns.
+ */
+class MNSConsumerDeleteMessageTest extends CamelTestSupport {
+
+ private final TestConfiguration testConfiguration = new
TestConfiguration();
+ private final MNSClient mnsClient = mock(MNSClient.class);
+ private final CloudQueue cloudQueue = mock(CloudQueue.class);
+ private final Queue<Message> queue = new ConcurrentLinkedQueue<>();
+ private final List<String> deletes = new CopyOnWriteArrayList<>();
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ CamelContext context = super.createCamelContext();
+
when(mnsClient.getQueueRef(testConfiguration.getProperty("queue"))).thenReturn(cloudQueue);
+ when(cloudQueue.popMessage()).thenAnswer(i -> queue.poll());
+ doAnswer(i ->
deletes.add(i.getArgument(0))).when(cloudQueue).deleteMessage(anyString());
+ context.getRegistry().bind("mnsClient", mnsClient);
+
+ Message message = new Message("hello");
+ message.setMessageId("m-1");
+ message.setReceiptHandle("handle-1");
+ queue.add(message);
+ return context;
+ }
+
+ @Test
+ void messageIsDeletedWhenTheRouteRemovesTheHeaders() throws Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("hello");
+ MNSConsumer consumer = startRoute("", route ->
route.removeHeaders("*"));
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(deletes).containsExactly("handle-1");
+ }
+
+ @Test
+ void messageIsDeletedWhenTheRouteReplacesTheMessage() throws Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("reply");
+ // the new message has none of the headers of the received one
+ MNSConsumer consumer = startRoute("", route -> route.process(exchange
-> {
+ DefaultMessage message = new DefaultMessage(exchange);
+ message.setBody("reply");
+ exchange.setMessage(message);
+ }));
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(deletes).containsExactly("handle-1");
+ }
+
+ @Test
+ void messageIsDeletedWithTheReceiptHandleItWasReceivedWith() throws
Exception {
+ // control: the route keeps the headers the consumer set
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("hello");
+ mock.expectedHeaderReceived(MNSHeaders.RECEIPT_HANDLE, "handle-1");
+ MNSConsumer consumer = startRoute("", route -> {
+ });
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(deletes).containsExactly("handle-1");
+ }
+
+ @Test
+ void messageIsNotDeletedWithoutDeleteAfterRead() throws Exception {
+ // control
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("hello");
+ MNSConsumer consumer = startRoute("&deleteAfterRead=false", route -> {
+ });
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(deletes).isEmpty();
+ }
+
+ private MNSConsumer startRoute(String options, Consumer<RouteDefinition>
steps) throws Exception {
+ String uri = "alibaba-mns:" + testConfiguration.getProperty("queue")
+ + "?region=" + testConfiguration.getProperty("region")
+ + "&accountEndpoint="
+ +
URLEncoder.encode(testConfiguration.getProperty("accountEndpoint"),
StandardCharsets.UTF_8)
+ + "&accessKey=" +
testConfiguration.getProperty("accessKey")
+ + "&secretKey=" +
testConfiguration.getProperty("secretKey")
+ + "&mnsClient=#mnsClient&startScheduler=false" + options;
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ RouteDefinition route = from(uri).routeId("mns");
+ steps.accept(route);
+ route.to("mock:result");
+ }
+ });
+ return (MNSConsumer) context.getRoute("mns").getConsumer();
+ }
+}
diff --git
a/components/camel-google/camel-google-storage/src/main/docs/google-storage-component.adoc
b/components/camel-google/camel-google-storage/src/main/docs/google-storage-component.adoc
index 4678ee29f9a0..338b384f679e 100644
---
a/components/camel-google/camel-google-storage/src/main/docs/google-storage-component.adoc
+++
b/components/camel-google/camel-google-storage/src/main/docs/google-storage-component.adoc
@@ -657,4 +657,8 @@ YAML::
In this case, the objects consumed will be moved to myCamelProcessedBucket
bucket and deleted from the original one (because of deleteAfterRead).
+The consumer deletes or moves the object it consumed, whatever the route does
with the `CamelGoogleCloudStorageObjectName`
+and `CamelGoogleCloudStorageBucketName` headers. A route can set the object
name header to write a copy of the object
+under another name.
+
diff --git
a/components/camel-google/camel-google-storage/src/main/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumer.java
b/components/camel-google/camel-google-storage/src/main/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumer.java
index 7c1db25cb0b9..90c98d051e3a 100644
---
a/components/camel-google/camel-google-storage/src/main/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumer.java
+++
b/components/camel-google/camel-google-storage/src/main/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumer.java
@@ -285,12 +285,14 @@ public class GoogleCloudStorageConsumer extends
ScheduledBatchPollingConsumer {
// update pending number of exchanges
pendingExchanges = total - index - 1;
+ // the object to delete or move once the exchange is done. It is
taken now, as the route may change or
+ // remove the object name header before it ends (the producer
takes the name of the object to write from it)
final String key =
exchange.getIn().getHeader(GoogleCloudStorageConstants.OBJECT_NAME,
String.class);
// add on completion to handle after work when the exchange is done
exchange.getExchangeExtension().addOnCompletion(new
Synchronization() {
public void onComplete(Exchange exchange) {
try {
- processCommit(exchange);
+ processCommit(exchange, key);
} finally {
inProgress.remove(key);
}
@@ -321,16 +323,27 @@ public class GoogleCloudStorageConsumer extends
ScheduledBatchPollingConsumer {
}
/**
- * Strategy to delete the message after being processed.
+ * Strategy to delete or move the object after being processed, the object
named by the
+ * {@link GoogleCloudStorageConstants#OBJECT_NAME} header.
*
- * @param exchange the exchange
+ * @param exchange the exchange
+ * @deprecated use {@link #processCommit(Exchange, String)},
which the consumer calls instead
*/
+ @Deprecated(since = "4.24.0")
protected void processCommit(Exchange exchange) {
+ processCommit(exchange,
exchange.getIn().getHeader(GoogleCloudStorageConstants.OBJECT_NAME,
String.class));
+ }
+
+ /**
+ * Strategy to delete or move the object after being processed.
+ *
+ * @param exchange the exchange
+ * @param key the name of the object the exchange was created from,
in the bucket of the endpoint
+ */
+ protected void processCommit(Exchange exchange, String key) {
+ final String bucketName = getConfiguration().getBucketName();
try {
if (getConfiguration().isMoveAfterRead()) {
- String bucketName =
exchange.getIn().getHeader(GoogleCloudStorageConstants.BUCKET_NAME,
String.class);
- String key =
exchange.getIn().getHeader(GoogleCloudStorageConstants.OBJECT_NAME,
String.class);
-
LOG.trace("Moving object from bucket {} with key {} to bucket
{}...", bucketName, key,
getConfiguration().getDestinationBucket());
@@ -343,9 +356,6 @@ public class GoogleCloudStorageConsumer extends
ScheduledBatchPollingConsumer {
getConfiguration().getDestinationBucket(),
copyWriter.getResult());
}
if (getConfiguration().isDeleteAfterRead()) {
- String bucketName =
exchange.getIn().getHeader(GoogleCloudStorageConstants.BUCKET_NAME,
String.class);
- String key =
exchange.getIn().getHeader(GoogleCloudStorageConstants.OBJECT_NAME,
String.class);
-
LOG.trace("Deleting object from bucket {} with key {}...",
bucketName, key);
boolean b = getStorageClient().delete(bucketName, key);
diff --git
a/components/camel-google/camel-google-storage/src/test/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumerDeleteMoveAfterReadTest.java
b/components/camel-google/camel-google-storage/src/test/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumerDeleteMoveAfterReadTest.java
new file mode 100644
index 000000000000..aa9420f9fecc
--- /dev/null
+++
b/components/camel-google/camel-google-storage/src/test/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumerDeleteMoveAfterReadTest.java
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.google.storage;
+
+import java.nio.charset.StandardCharsets;
+import java.util.function.Consumer;
+
+import com.google.cloud.storage.Blob;
+import com.google.cloud.storage.BlobInfo;
+import com.google.cloud.storage.BucketInfo;
+import com.google.cloud.storage.Storage;
+import org.apache.camel.CamelContext;
+import org.apache.camel.builder.RouteBuilder;
+import
org.apache.camel.component.google.storage.localstorage.LocalStorageHelper;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.model.RouteDefinition;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+
+import static org.apache.camel.builder.Builder.simple;
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * With deleteAfterRead or moveAfterRead the consumer deletes or moves the
object it consumed once the exchange is done,
+ * whatever the route did with the CamelGoogleCloudStorageObjectName and
CamelGoogleCloudStorageBucketName headers in
+ * the meantime: a route may set the object name to write a copy of the object
under another name (the producer takes
+ * the name from that header), or remove the Camel headers before it calls
another system.
+ * <p>
+ * The scheduler of the consumer is not started: the tests call {@code poll()}
themselves. The routes are synchronous,
+ * so the delete or the move is done when {@code poll()} returns.
+ */
+class GoogleCloudStorageConsumerDeleteMoveAfterReadTest extends
CamelTestSupport {
+
+ private Storage storage;
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ CamelContext context = super.createCamelContext();
+ storage = LocalStorageHelper.getOptions().getService();
+ storage.create(BucketInfo.of("in"));
+ storage.create(BucketInfo.of("archive"));
+ storage.create(BlobInfo.newBuilder("in", "todo/a.txt").build(),
"hello".getBytes(StandardCharsets.UTF_8));
+ context.getComponent("google-storage",
GoogleCloudStorageComponent.class).getConfiguration()
+ .setStorageClient(storage);
+ return context;
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedObjectWhenTheRouteRenamesIt() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ GoogleCloudStorageConsumer consumer = startRoute("", this::writeCopy);
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(exists("in", "todo/a.txt")).as("the consumed object is
deleted").isFalse();
+ assertThat(exists("in", "out/todo/a.txt")).as("the copy the route
wrote is kept").isTrue();
+ assertThat(consumer.poll()).as("the next poll does not consume the
object again").isZero();
+ }
+
+ @Test
+ void
deleteAfterReadDeletesTheConsumedObjectWhenTheRouteRemovesTheHeaders() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ GoogleCloudStorageConsumer consumer
+ = startRoute("", route ->
route.removeHeaders("CamelGoogleCloudStorage*"));
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(exists("in", "todo/a.txt")).as("the consumed object is
deleted").isFalse();
+ assertThat(consumer.poll()).as("the next poll does not consume the
object again").isZero();
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedObjectWhenTheRouteRenamesIt() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ GoogleCloudStorageConsumer consumer =
startRoute("&moveAfterRead=true&destinationBucket=archive", this::writeCopy);
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(exists("archive", "todo/a.txt")).as("the consumed object is
copied to the destination bucket").isTrue();
+ assertThat(exists("archive", "out/todo/a.txt")).as("the copy the route
wrote is not moved").isFalse();
+ assertThat(exists("in", "todo/a.txt")).as("the consumed object is
deleted").isFalse();
+ assertThat(exists("in", "out/todo/a.txt")).as("the copy the route
wrote is kept").isTrue();
+ assertThat(consumer.poll()).as("the next poll does not consume the
object again").isZero();
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedObject() throws Exception {
+ // control: the route keeps the headers the consumer set
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ mock.expectedHeaderReceived(GoogleCloudStorageConstants.OBJECT_NAME,
"todo/a.txt");
+ mock.expectedHeaderReceived(GoogleCloudStorageConstants.BUCKET_NAME,
"in");
+ GoogleCloudStorageConsumer consumer = startRoute("", route -> {
+ });
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(exists("in", "todo/a.txt")).isFalse();
+ assertThat(consumer.poll()).isZero();
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedObject() throws Exception {
+ // control: the route keeps the headers the consumer set
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ GoogleCloudStorageConsumer consumer =
startRoute("&moveAfterRead=true&destinationBucket=archive", route -> {
+ });
+
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(exists("archive", "todo/a.txt")).isTrue();
+ assertThat(exists("in", "todo/a.txt")).isFalse();
+ }
+
+ @Test
+ void objectIsKeptWithoutDeleteAfterRead() throws Exception {
+ // control: without deleteAfterRead the object stays, and is consumed
again
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(2);
+ GoogleCloudStorageConsumer consumer =
startRoute("&deleteAfterRead=false", route -> {
+ });
+
+ assertThat(consumer.poll()).isEqualTo(1);
+ assertThat(consumer.poll()).isEqualTo(1);
+
+ mock.assertIsSatisfied();
+ assertThat(exists("in", "todo/a.txt")).isTrue();
+ }
+
+ /**
+ * Writes a copy of the object to out/ in the same bucket: the producer
takes the name of the object from the
+ * header.
+ */
+ private void writeCopy(RouteDefinition route) {
+ route.setHeader(GoogleCloudStorageConstants.OBJECT_NAME,
+ simple("out/${header." +
GoogleCloudStorageConstants.OBJECT_NAME + "}"))
+ .to("google-storage://in");
+ }
+
+ private GoogleCloudStorageConsumer startRoute(String options,
Consumer<RouteDefinition> steps) throws Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ RouteDefinition route =
from("google-storage://in?prefix=todo/&startScheduler=false" + options)
+ .routeId("gcs");
+ steps.accept(route);
+ route.to("mock:result");
+ }
+ });
+ return (GoogleCloudStorageConsumer)
context.getRoute("gcs").getConsumer();
+ }
+
+ private boolean exists(String bucket, String name) {
+ Blob blob = storage.get(bucket, name);
+ return blob != null && blob.exists();
+ }
+}
diff --git a/components/camel-minio/src/main/docs/minio-component.adoc
b/components/camel-minio/src/main/docs/minio-component.adoc
index 97527f3c21a9..6e5dc2d0c42f 100644
--- a/components/camel-minio/src/main/docs/minio-component.adoc
+++ b/components/camel-minio/src/main/docs/minio-component.adoc
@@ -692,6 +692,9 @@ YAML::
In this case, the objects consumed will be moved to `myothercamelbucket`
bucket and deleted from the original one (because of `deleteAfterRead` set to
true as default).
+The consumer deletes or moves the object it consumed, whatever the route does
with the `CamelMinioObjectName` and
+`CamelMinioBucketName` headers. A route can set the object name header to
write a copy of the object under another name.
+
=== Using a POJO as body
Sometimes build a Minio Request can be complex because of multiple options.
diff --git
a/components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
b/components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
index 977e4473752b..4b39a76e76f0 100644
---
a/components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
+++
b/components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java
@@ -398,6 +398,9 @@ public class MinioConsumer extends
ScheduledBatchPollingConsumer {
// update pending number of exchanges
pendingExchanges = total - index - 1;
+ // the object to get, and to delete or move once the exchange is
done. It is taken now, as the route may
+ // change or remove the object name header before it ends (the
producer takes the name of the object to
+ // write from it)
String srcBucketName =
exchange.getIn().getHeader(MinioConstants.BUCKET_NAME, String.class);
String srcObjectName =
exchange.getIn().getHeader(MinioConstants.OBJECT_NAME, String.class);
if (getConfiguration().isIncludeBody()) {
@@ -443,7 +446,7 @@ public class MinioConsumer extends
ScheduledBatchPollingConsumer {
exchange.getExchangeExtension().addOnCompletion(new
Synchronization() {
public void onComplete(Exchange exchange) {
try {
- processCommit(exchange);
+ processCommit(exchange, srcObjectName);
} finally {
inProgress.remove(srcObjectName);
}
@@ -487,15 +490,26 @@ public class MinioConsumer extends
ScheduledBatchPollingConsumer {
}
/**
- * Strategy to delete the message after being processed.
+ * Strategy to delete or move the object after being processed, the object
named by the
+ * {@link MinioConstants#OBJECT_NAME} header.
*
- * @param exchange the exchange
+ * @param exchange the exchange
+ * @deprecated use {@link #processCommit(Exchange, String)},
which the consumer calls instead
*/
+ @Deprecated(since = "4.24.0")
protected void processCommit(Exchange exchange) {
- try {
- String srcBucketName =
exchange.getIn().getHeader(MinioConstants.BUCKET_NAME, String.class);
- String srcObjectName =
exchange.getIn().getHeader(MinioConstants.OBJECT_NAME, String.class);
+ processCommit(exchange,
exchange.getIn().getHeader(MinioConstants.OBJECT_NAME, String.class));
+ }
+ /**
+ * Strategy to delete or move the object after being processed.
+ *
+ * @param exchange the exchange
+ * @param srcObjectName the name of the object the exchange was created
from, in the bucket of the endpoint
+ */
+ protected void processCommit(Exchange exchange, String srcObjectName) {
+ final String srcBucketName = getConfiguration().getBucketName();
+ try {
if (getConfiguration().isDeleteAfterRead() ||
getConfiguration().isMoveAfterRead()) {
if (getConfiguration().isMoveAfterRead()) {
copyObject(srcBucketName, srcObjectName);
diff --git
a/components/camel-minio/src/test/java/org/apache/camel/component/minio/FakeS3Server.java
b/components/camel-minio/src/test/java/org/apache/camel/component/minio/FakeS3Server.java
index 2cc55c17c5e7..7028976d76d3 100644
---
a/components/camel-minio/src/test/java/org/apache/camel/component/minio/FakeS3Server.java
+++
b/components/camel-minio/src/test/java/org/apache/camel/component/minio/FakeS3Server.java
@@ -24,12 +24,14 @@ import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.HashMap;
+import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.NavigableMap;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import com.sun.net.httpserver.HttpExchange;
@@ -37,8 +39,9 @@ import com.sun.net.httpserver.HttpServer;
/**
* A minimal in-process fake of the S3 API calls the minio consumer and
producer make for a single bucket: head bucket,
- * list objects (v2, paginated with max-keys and
start-after/continuation-token), stat, get (also of a byte range), put
- * and delete object. The gets (or the stats and gets) of an object can be
refused with 403 (AccessDenied).
+ * list objects (v2, with a prefix, paginated with max-keys and
start-after/continuation-token), stat, get (also of a
+ * byte range), put, copy and delete object. A copy to another bucket is
answered and recorded, but the other bucket
+ * holds nothing. The gets (or the stats and gets) of an object can be refused
with 403 (AccessDenied).
*/
final class FakeS3Server {
@@ -51,6 +54,8 @@ final class FakeS3Server {
private final Set<String> deleteWhenStatted =
ConcurrentHashMap.newKeySet();
private final Set<String> refusedGets = ConcurrentHashMap.newKeySet();
private final Set<String> refusedStats = ConcurrentHashMap.newKeySet();
+ private final List<String> copies = new CopyOnWriteArrayList<>();
+ private final List<String> deletes = new CopyOnWriteArrayList<>();
FakeS3Server(String bucket) throws IOException {
this.bucket = bucket;
@@ -126,10 +131,41 @@ final class FakeS3Server {
return n != null ? n.get() : 0;
}
+ /**
+ * The copies made, in order, as {@code source-bucket/key ->
destination-bucket/key}.
+ */
+ List<String> copies() {
+ return copies;
+ }
+
+ /**
+ * The keys of the objects deleted, in order, whether they existed or not.
+ */
+ List<String> deletes() {
+ return deletes;
+ }
+
private void handle(HttpExchange http) throws IOException {
- String path = http.getRequestURI().getRawPath();
- String key = path.length() > bucket.length() + 2 ?
decode(path.substring(bucket.length() + 2)) : null;
+ // path style: /bucket or /bucket/key
+ String path = http.getRequestURI().getRawPath().substring(1);
+ int slash = path.indexOf('/');
+ String pathBucket = decode(slash < 0 ? path : path.substring(0,
slash));
+ String key = slash >= 0 && slash < path.length() - 1 ?
decode(path.substring(slash + 1)) : null;
String method = http.getRequestMethod();
+ String copySource =
http.getRequestHeaders().getFirst("x-amz-copy-source");
+ if (key != null && "PUT".equals(method) && copySource != null) {
+ copy(http, copySource, pathBucket, key);
+ return;
+ }
+ if (!bucket.equals(pathBucket)) {
+ // another bucket, such as the destination bucket of a move: it
exists, and holds nothing
+ if (key == null) {
+ respond(http, 200, null, null);
+ } else {
+ noSuchKey(http, key);
+ }
+ return;
+ }
if (key == null) {
if ("GET".equals(method) && query(http).containsKey("list-type")) {
list(http, query(http));
@@ -197,6 +233,7 @@ final class FakeS3Server {
http.close();
}
case "DELETE" -> {
+ deletes.add(key);
objects.remove(key);
http.sendResponseHeaders(204, -1);
http.close();
@@ -205,8 +242,31 @@ final class FakeS3Server {
}
}
+ private void copy(HttpExchange http, String copySource, String toBucket,
String toKey) throws IOException {
+ // x-amz-copy-source: /bucket/key, encoded as a path
+ String source = decode(copySource.startsWith("/") ?
copySource.substring(1) : copySource);
+ int slash = source.indexOf('/');
+ String fromBucket = source.substring(0, slash);
+ String fromKey = source.substring(slash + 1);
+ copies.add(fromBucket + "/" + fromKey + " -> " + toBucket + "/" +
toKey);
+ byte[] data = bucket.equals(fromBucket) ? objects.get(fromKey) : null;
+ if (data == null) {
+ noSuchKey(http, fromKey);
+ return;
+ }
+ if (bucket.equals(toBucket)) {
+ objects.put(toKey, data);
+ }
+ String xml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
+ + "<CopyObjectResult
xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">"
+ + "<LastModified>2026-10-09T00:00:00.000Z</LastModified>"
+ +
"<ETag>\"0123456789abcdef0123456789abcdef\"</ETag></CopyObjectResult>";
+ respond(http, 200, "application/xml", xml);
+ }
+
private void list(HttpExchange http, Map<String, String> query) throws
IOException {
int maxKeys = query.containsKey("max-keys") ?
Integer.parseInt(query.get("max-keys")) : 1000;
+ String prefix = query.getOrDefault("prefix", "");
String after = query.get("continuation-token");
if (after == null) {
after = query.get("start-after");
@@ -216,6 +276,9 @@ final class FakeS3Server {
int count = 0;
String last = null;
for (Map.Entry<String, byte[]> e : view.entrySet()) {
+ if (!e.getKey().startsWith(prefix)) {
+ continue;
+ }
if (count == maxKeys) {
break;
}
@@ -232,7 +295,11 @@ final class FakeS3Server {
objects.remove(key);
}
}
- boolean truncated = last != null && objects.higherKey(last) != null;
+ String next = last != null ? objects.higherKey(last) : null;
+ while (next != null && !next.startsWith(prefix)) {
+ next = objects.higherKey(next);
+ }
+ boolean truncated = next != null;
String xml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
+ "<ListBucketResult
xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">"
+ "<Name>" + bucket + "</Name><Prefix></Prefix>"
diff --git
a/components/camel-minio/src/test/java/org/apache/camel/component/minio/MinioConsumerDeleteMoveAfterReadTest.java
b/components/camel-minio/src/test/java/org/apache/camel/component/minio/MinioConsumerDeleteMoveAfterReadTest.java
new file mode 100644
index 000000000000..ec97e94d5475
--- /dev/null
+++
b/components/camel-minio/src/test/java/org/apache/camel/component/minio/MinioConsumerDeleteMoveAfterReadTest.java
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.minio;
+
+import java.util.List;
+import java.util.function.Consumer;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.model.RouteDefinition;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+import static org.apache.camel.builder.Builder.simple;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * With deleteAfterRead or moveAfterRead the consumer deletes or moves the
object it consumed once the exchange is done,
+ * whatever the route did with the CamelMinioObjectName and
CamelMinioBucketName headers in the meantime: a route may
+ * set the object name to write a copy of the object under another name (the
producer takes the name from that header),
+ * or remove the Camel headers before it calls another system.
+ * <p>
+ * The scheduler of the consumer is not started: the tests call {@code poll()}
themselves. The routes are synchronous,
+ * so the delete or the move is done when {@code poll()} returns.
+ */
+class MinioConsumerDeleteMoveAfterReadTest extends CamelTestSupport {
+
+ private FakeS3Server s3;
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ s3 = new FakeS3Server("inbox");
+ s3.put("todo/a.txt", "hello");
+ s3.start();
+ return super.createCamelContext();
+ }
+
+ @AfterEach
+ void stopServer() {
+ if (s3 != null) {
+ s3.stop();
+ }
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedObjectWhenTheRouteRenamesIt() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ MinioConsumer consumer = startRoute("", this::writeCopy);
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("todo/a.txt"), s3.deletes());
+ assertTrue(s3.contains("out/todo/a.txt"), "the copy the route wrote is
kept");
+ assertEquals(0, consumer.poll(), "the next poll does not consume the
object again");
+ }
+
+ @Test
+ void
deleteAfterReadDeletesTheConsumedObjectWhenTheRouteRemovesTheHeaders() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ MinioConsumer consumer = startRoute("", route ->
route.removeHeaders("CamelMinio*"));
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("todo/a.txt"), s3.deletes());
+ assertEquals(0, consumer.poll(), "the next poll does not consume the
object again");
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedObjectWhenTheRouteRenamesIt() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ MinioConsumer consumer =
startRoute("&moveAfterRead=true&destinationBucketName=archive",
this::writeCopy);
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("inbox/todo/a.txt -> archive/todo/a.txt"),
s3.copies());
+ assertEquals(List.of("todo/a.txt"), s3.deletes());
+ assertTrue(s3.contains("out/todo/a.txt"), "the copy the route wrote is
kept");
+ assertEquals(0, consumer.poll(), "the next poll does not consume the
object again");
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedObject() throws Exception {
+ // control: the route keeps the headers the consumer set
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ mock.expectedHeaderReceived(MinioConstants.OBJECT_NAME, "todo/a.txt");
+ mock.expectedHeaderReceived(MinioConstants.BUCKET_NAME, "inbox");
+ MinioConsumer consumer = startRoute("", route -> {
+ });
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("todo/a.txt"), s3.deletes());
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedObject() throws Exception {
+ // control: the destination name is the configured one
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ MinioConsumer consumer = startRoute(
+
"&moveAfterRead=true&destinationBucketName=archive&destinationObjectName=done.txt",
route -> {
+ });
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("inbox/todo/a.txt -> archive/done.txt"),
s3.copies());
+ assertEquals(List.of("todo/a.txt"), s3.deletes());
+ }
+
+ @Test
+ void objectIsKeptWithoutDeleteAfterRead() throws Exception {
+ // control: without deleteAfterRead the object stays, and is consumed
again
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(2);
+ MinioConsumer consumer = startRoute("&deleteAfterRead=false", route ->
{
+ });
+
+ assertEquals(1, consumer.poll());
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of(), s3.deletes());
+ assertEquals(List.of(), s3.copies());
+ }
+
+ /**
+ * Writes a copy of the object to out/ in the same bucket: the producer
takes the name of the object from the
+ * header.
+ */
+ private void writeCopy(RouteDefinition route) {
+ route.setHeader(MinioConstants.OBJECT_NAME, simple("out/${header." +
MinioConstants.OBJECT_NAME + "}"))
+ .to("minio://inbox?" + credentials());
+ }
+
+ private MinioConsumer startRoute(String options, Consumer<RouteDefinition>
steps) throws Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ RouteDefinition route = from("minio://inbox?" + credentials()
+ "&prefix=todo/&startScheduler=false" + options)
+ .routeId("minio");
+ steps.accept(route);
+ route.to("mock:result");
+ }
+ });
+ return (MinioConsumer) context.getRoute("minio").getConsumer();
+ }
+
+ private String credentials() {
+ return "accessKey=xxx&secretKey=yyy®ion=us-east-1&endpoint=" +
s3.endpoint();
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index 20595e730ecf..6044bcad5c57 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -6600,6 +6600,23 @@ poll. The failure is given to the exception handler of
the consumer, with the ob
whole bucket, one such object stopped the consumer from consuming any object.
The poll still fails when none of the
listed objects could be got.
+=== camel-google-storage, camel-minio, camel-alibaba-mns - the consumer
deletes or moves what it consumed
+
+With `deleteAfterRead` or `moveAfterRead`, the `google-storage` and `minio`
consumers now delete or move the object
+they consumed, in the bucket of the endpoint. Before, they took the name and
the bucket of the object from the
+`CamelGoogleCloudStorageObjectName` and `CamelGoogleCloudStorageBucketName`
headers, or from the `CamelMinioObjectName`
+and `CamelMinioBucketName` headers, as the route left them. A route that set
the object name header, for example to
+write a copy of the object under another name, deleted or moved that copy, and
the consumed object was consumed again
+by the next poll. A route that removed these headers deleted nothing.
+
+The `alibaba-mns` consumer now deletes the message with the receipt handle it
received it with. Before, it took the
+`CamelAlibabaMnsReceiptHandle` header as the route left it, so a route that
removed the headers or replaced the message
+did not delete it, and the message was received again once it was visible
again.
+
+The consumers now call `processCommit(Exchange, String)` with the name of the
object or the receipt handle. A subclass
+of `GoogleCloudStorageConsumer`, `MinioConsumer` or `MNSConsumer` that
overrides `processCommit(Exchange)` must override
+the new method instead. `processCommit(Exchange)` is deprecated.
+
=== camel-mongodb-gridfs - the consumer routes a file once, offers the file of
a failed exchange again and sets CamelFileName, and an unknown operation fails
With the `TimeStamp` (default) and `PersistentTimestamp` query strategies, the
`mongodb-gridfs` consumer routed