This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 2e2bb4771ed8 CAMEL-27015: camel-aws2-s3, camel-azure-storage-blob,
camel-aws2-sqs - the consumers delete or move the object they consumed,
whatever the route did with its headers (#30168)
2e2bb4771ed8 is described below
commit 2e2bb4771ed886f173f07fe5ab60012b9f8f8424
Author: Federico Mariani <[email protected]>
AuthorDate: Mon Nov 16 16:08:58 2026 +0100
CAMEL-27015: camel-aws2-s3, camel-azure-storage-blob, camel-aws2-sqs - the
consumers delete or move the object they consumed, whatever the route did with
its headers (#30168)
- With `deleteAfterRead` or `moveAfterRead`, the `aws2-s3` and
`azure-storage-blob` consumers took the key, the bucket and the blob name from
the headers when the exchange completed, and `aws2-sqs` took the receipt handle
from its header. A route that set these headers (for example to write a copy of
the object under another name) deleted or moved that copy. A route that removed
them, or replaced the message, left the object or the message in place, and it
was consumed again.
- The consumers now take the key (from the exchange property the S3
consumer sets), the blob name and the receipt handle before the exchange is
routed, and pass them to a new `processCommit(Exchange, String)`. The S3 move
copies from the bucket of the endpoint, as the delete already did.
`processCommit(Exchange)` is kept for subclasses and deprecated, since the
consumers no longer call it. The WARN of a blob without a name no longer says
that the name is missing from the headers.
- `AWS2S3Consumer.processCommit` catches `SdkException` instead of
`AwsServiceException`, so a failure raised by the client itself reaches the
exception handler of the consumer.
- Same approach as CAMEL-25945 for `azure-storage-queue`.
_Claude Code on behalf of Federico Mariani_
Co-Authored-By: Claude <[email protected]>
---
.../camel/component/aws2/s3/AWS2S3Consumer.java | 37 +--
.../s3/AWS2S3ConsumerDeleteMoveAfterReadTest.java | 200 ++++++++++++++++
.../camel/component/aws2/s3/FakeS3Server.java | 57 ++++-
.../camel/component/aws2/sqs/Sqs2Consumer.java | 23 +-
.../aws2/sqs/SqsConsumerDeleteMessageTest.java | 105 ++++++++
.../component/azure/storage/blob/BlobConsumer.java | 22 +-
.../blob/BlobConsumerDeleteMoveAfterReadTest.java | 265 +++++++++++++++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 20 ++
8 files changed, 702 insertions(+), 27 deletions(-)
diff --git
a/components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java
b/components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java
index 18c778104f87..b4ce3c482970 100644
---
a/components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java
+++
b/components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java
@@ -44,6 +44,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.awscore.exception.AwsServiceException;
import software.amazon.awssdk.core.ResponseInputStream;
+import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
@@ -401,12 +402,15 @@ public class AWS2S3Consumer extends
ScheduledBatchPollingConsumer {
exchange.setProperty(ExchangePropertyKey.BATCH_SIZE, total);
exchange.setProperty(ExchangePropertyKey.BATCH_COMPLETE, index ==
total - 1);
+ // the object to delete or move once the exchange is done. It is
taken now, as the route may change or
+ // remove the key and bucket headers before it ends (the producer
takes the key of the object to write from
+ // the key header)
+ final String key = exchange.getProperty(AWS2S3Constants.KEY,
String.class);
// add on completion to handle after work when the exchange is done
exchange.getExchangeExtension().addOnCompletion(new
Synchronization() {
public void onComplete(Exchange exchange) {
- final String key =
exchange.getProperty(AWS2S3Constants.KEY, String.class);
try {
- processCommit(exchange);
+ processCommit(exchange, key);
} finally {
if (key != null) {
getEndpoint().getInProgressRepository().remove(key);
@@ -415,7 +419,6 @@ public class AWS2S3Consumer extends
ScheduledBatchPollingConsumer {
}
public void onFailure(Exchange exchange) {
- final String key =
exchange.getProperty(AWS2S3Constants.KEY, String.class);
try {
processRollback(exchange);
} finally {
@@ -481,16 +484,26 @@ public class AWS2S3Consumer extends
ScheduledBatchPollingConsumer {
}
/**
- * Strategy to delete the message after being processed.
+ * Strategy to delete or move the object after being processed, the object
the exchange was created from.
*
- * @param exchange the exchange
+ * @param exchange the exchange
+ * @deprecated use {@link #processCommit(Exchange, String)},
which the consumer calls instead
*/
+ @Deprecated(since = "4.24.0")
protected void processCommit(Exchange exchange) {
+ processCommit(exchange, exchange.getProperty(AWS2S3Constants.KEY,
String.class));
+ }
+
+ /**
+ * Strategy to delete or move the object after being processed.
+ *
+ * @param exchange the exchange
+ * @param key the key of the object the exchange was created from, in
the bucket of the endpoint
+ */
+ protected void processCommit(Exchange exchange, String key) {
+ final String bucketName = getConfiguration().getBucketName();
try {
if (getConfiguration().isMoveAfterRead()) {
- String bucketName =
exchange.getIn().getHeader(AWS2S3Constants.BUCKET_NAME, String.class);
- String key = exchange.getIn().getHeader(AWS2S3Constants.KEY,
String.class);
-
LOG.trace("Moving object from bucket {} with key {} to bucket
{}...", bucketName, key,
getConfiguration().getDestinationBucket());
@@ -517,17 +530,13 @@ public class AWS2S3Consumer extends
ScheduledBatchPollingConsumer {
getConfiguration().getDestinationBucket());
}
if (getConfiguration().isDeleteAfterRead()) {
- String bucketName =
exchange.getIn().getHeader(AWS2S3Constants.BUCKET_NAME, String.class);
- String key = exchange.getIn().getHeader(AWS2S3Constants.KEY,
String.class);
-
LOG.trace("Deleting object from bucket {} with key {}...",
bucketName, key);
- getAmazonS3Client().deleteObject(
-
DeleteObjectRequest.builder().bucket(getConfiguration().getBucketName()).key(key).build());
+
getAmazonS3Client().deleteObject(DeleteObjectRequest.builder().bucket(bucketName).key(key).build());
LOG.trace("Deleted object from bucket {} with key {}...",
bucketName, key);
}
- } catch (AwsServiceException e) {
+ } catch (SdkException e) {
getExceptionHandler().handleException("Error occurred during
moving or deleting object. This exception is ignored.",
exchange, e);
}
diff --git
a/components/camel-aws/camel-aws2-s3/src/test/java/org/apache/camel/component/aws2/s3/AWS2S3ConsumerDeleteMoveAfterReadTest.java
b/components/camel-aws/camel-aws2-s3/src/test/java/org/apache/camel/component/aws2/s3/AWS2S3ConsumerDeleteMoveAfterReadTest.java
new file mode 100644
index 000000000000..b2e3e78ff5ef
--- /dev/null
+++
b/components/camel-aws/camel-aws2-s3/src/test/java/org/apache/camel/component/aws2/s3/AWS2S3ConsumerDeleteMoveAfterReadTest.java
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.aws2.s3;
+
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.function.Consumer;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.model.RouteDefinition;
+import org.apache.camel.spi.ExceptionHandler;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import software.amazon.awssdk.core.exception.SdkClientException;
+
+import static org.apache.camel.builder.Builder.constant;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+
+/**
+ * With deleteAfterRead or moveAfterRead the consumer deletes or moves the
object it consumed once the exchange is done,
+ * whatever the route did with the CamelAwsS3Key and CamelAwsS3BucketName
headers in the meantime: a route may set them
+ * to write a copy of the object under another key, or remove the Camel
headers before it calls another system.
+ * <p>
+ * The scheduler of the consumer is not started: the tests call {@code poll()}
themselves. The routes are synchronous,
+ * so the delete or the move is done when {@code poll()} returns.
+ */
+class AWS2S3ConsumerDeleteMoveAfterReadTest extends CamelTestSupport {
+
+ private FakeS3Server s3;
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ s3 = new FakeS3Server("bkt");
+ s3.start();
+ return super.createCamelContext();
+ }
+
+ @AfterEach
+ void stopServer() {
+ if (s3 != null) {
+ s3.stop();
+ }
+ }
+
+ @Test
+ void
deleteAfterReadDeletesTheConsumedObjectWhenTheRouteChangesTheKeyHeader() throws
Exception {
+ s3.put("a.txt", "a");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("a");
+ // for example the key under which the route writes a copy of the
object
+ AWS2S3Consumer consumer = startRoute("", route ->
route.setHeader(AWS2S3Constants.KEY, constant("out/a.txt")));
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("a.txt"), s3.deletes());
+ // so the next poll does not consume it again
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void
deleteAfterReadDeletesTheConsumedObjectWhenTheRouteRemovesTheHeaders() throws
Exception {
+ s3.put("a.txt", "a");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("a");
+ AWS2S3Consumer consumer = startRoute("", route ->
route.removeHeaders("CamelAwsS3*"));
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("a.txt"), s3.deletes());
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedObjectWhenTheRouteChangesTheHeaders()
throws Exception {
+ s3.put("a.txt", "a");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("a");
+ AWS2S3Consumer consumer =
startRoute("&moveAfterRead=true&destinationBucket=archive",
+ route -> route.setHeader(AWS2S3Constants.BUCKET_NAME,
constant("other"))
+ .setHeader(AWS2S3Constants.KEY,
constant("out/a.txt")));
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("bkt/a.txt -> archive/a.txt"), s3.copies());
+ assertEquals(List.of("a.txt"), s3.deletes());
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedObject() throws Exception {
+ // control: the route keeps the headers the consumer set
+ s3.put("a.txt", "a");
+ s3.put("b.txt", "b");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("a", "b");
+ AWS2S3Consumer consumer = startRoute("", route -> {
+ });
+
+ assertEquals(2, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("a.txt", "b.txt"), s3.deletes());
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedObject() throws Exception {
+ // control: the destination key is built from the key of the consumed
object
+ s3.put("a.txt", "a");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("a");
+ AWS2S3Consumer consumer = startRoute(
+
"&moveAfterRead=true&destinationBucket=archive&destinationBucketPrefix=done/&destinationBucketSuffix=.bak",
+ route -> {
+ });
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("bkt/a.txt -> archive/done/a.txt.bak"),
s3.copies());
+ assertEquals(List.of("a.txt"), s3.deletes());
+ }
+
+ @Test
+ void deleteThatCannotBeSentIsReportedToTheExceptionHandler() throws
Exception {
+ s3.put("a.txt", "a");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("a");
+ // S3 cannot be reached when the exchange is done: the client fails
without an answer from S3
+ AWS2S3Consumer consumer = startRoute("", route ->
route.process(exchange -> s3.stop()));
+ RecordingExceptionHandler handler = new RecordingExceptionHandler();
+ consumer.setExceptionHandler(handler);
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("Error occurred during moving or deleting object.
This exception is ignored."),
+ handler.messages);
+ assertInstanceOf(SdkClientException.class, handler.exceptions.get(0));
+ }
+
+ private AWS2S3Consumer startRoute(String options,
Consumer<RouteDefinition> steps) throws Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ RouteDefinition route
+ = from(
+
"aws2-s3://bkt?accessKey=xxx&secretKey=yyy®ion=us-east-1&overrideEndpoint=true"
+ +
"&forcePathStyle=true&uriEndpointOverride=" + s3.endpoint()
+ + "&startScheduler=false" +
options)
+ .routeId("s3");
+ steps.accept(route);
+ route.convertBodyTo(String.class).to("mock:result");
+ }
+ });
+ return (AWS2S3Consumer) context.getRoute("s3").getConsumer();
+ }
+
+ private static final class RecordingExceptionHandler implements
ExceptionHandler {
+ final List<String> messages = new CopyOnWriteArrayList<>();
+ final List<Throwable> exceptions = new CopyOnWriteArrayList<>();
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleException(null, null, exception);
+ }
+
+ @Override
+ public void handleException(String message, Throwable exception) {
+ handleException(message, null, exception);
+ }
+
+ @Override
+ public void handleException(String message, Exchange exchange,
Throwable exception) {
+ messages.add(String.valueOf(message));
+ exceptions.add(exception);
+ }
+ }
+}
diff --git
a/components/camel-aws/camel-aws2-s3/src/test/java/org/apache/camel/component/aws2/s3/FakeS3Server.java
b/components/camel-aws/camel-aws2-s3/src/test/java/org/apache/camel/component/aws2/s3/FakeS3Server.java
index 0cee9c222835..48d7a820f4f0 100644
---
a/components/camel-aws/camel-aws2-s3/src/test/java/org/apache/camel/component/aws2/s3/FakeS3Server.java
+++
b/components/camel-aws/camel-aws2-s3/src/test/java/org/apache/camel/component/aws2/s3/FakeS3Server.java
@@ -29,6 +29,7 @@ import java.util.NavigableMap;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import com.sun.net.httpserver.HttpExchange;
@@ -36,9 +37,9 @@ import com.sun.net.httpserver.HttpServer;
/**
* A minimal in-process fake of the S3 API calls the consumer makes for a
single bucket (path-style requests): head
- * bucket, list objects (v2, paginated with max-keys and
start-after/continuation-token), head, get and delete object.
- * An object can be removed right after it was listed, the gets (and heads) of
an object can be refused, and the body of
- * an object can be cut short.
+ * bucket, list objects (v2, paginated with max-keys and
start-after/continuation-token), head, get, delete and copy
+ * object. A copy to another bucket is only recorded. An object can be removed
right after it was listed, the gets (and
+ * heads) of an object can be refused, and the body of an object can be cut
short.
*/
final class FakeS3Server {
@@ -47,6 +48,8 @@ final class FakeS3Server {
private final NavigableMap<String, byte[]> objects = new
ConcurrentSkipListMap<>();
private final Map<String, AtomicInteger> gets = new ConcurrentHashMap<>();
private final Map<String, AtomicInteger> heads = new ConcurrentHashMap<>();
+ private final List<String> deletes = new CopyOnWriteArrayList<>();
+ private final List<String> copies = new CopyOnWriteArrayList<>();
private final Set<String> removedWhenListed =
ConcurrentHashMap.newKeySet();
private final Set<String> refused = ConcurrentHashMap.newKeySet();
private final Map<String, AtomicInteger> refusedGets = new
ConcurrentHashMap<>();
@@ -128,9 +131,26 @@ final class FakeS3Server {
return n != null ? n.get() : 0;
}
+ /**
+ * The keys of the delete object requests, in order.
+ */
+ List<String> deletes() {
+ return List.copyOf(deletes);
+ }
+
+ /**
+ * The copy object requests, in order, as {@code sourceBucket/sourceKey ->
destinationBucket/destinationKey}.
+ */
+ List<String> copies() {
+ return List.copyOf(copies);
+ }
+
private void handle(HttpExchange http) throws IOException {
- String path = http.getRequestURI().getRawPath();
- String key = path.length() > bucket.length() + 2 ?
decode(path.substring(bucket.length() + 2)) : null;
+ // path-style requests: /bucket/key
+ String path = http.getRequestURI().getRawPath().substring(1);
+ int slash = path.indexOf('/');
+ String requestBucket = decode(slash >= 0 ? path.substring(0, slash) :
path);
+ String key = slash >= 0 && slash < path.length() - 1 ?
decode(path.substring(slash + 1)) : null;
String method = http.getRequestMethod();
if (key == null) {
if ("GET".equals(method) && query(http).containsKey("list-type")) {
@@ -141,6 +161,11 @@ final class FakeS3Server {
}
return;
}
+ String copySource =
http.getRequestHeaders().getFirst("x-amz-copy-source");
+ if ("PUT".equals(method) && copySource != null) {
+ copy(http, decode(copySource), requestBucket, key);
+ return;
+ }
byte[] data = objects.get(key);
switch (method) {
case "HEAD" -> {
@@ -181,6 +206,7 @@ final class FakeS3Server {
}
}
case "DELETE" -> {
+ deletes.add(key);
objects.remove(key);
http.sendResponseHeaders(204, -1);
http.close();
@@ -189,6 +215,27 @@ final class FakeS3Server {
}
}
+ private void copy(HttpExchange http, String source, String
destinationBucket, String destinationKey)
+ throws IOException {
+ String sourcePath = source.startsWith("/") ? source.substring(1) :
source;
+ int slash = sourcePath.indexOf('/');
+ String sourceBucket = sourcePath.substring(0, slash);
+ String sourceKey = sourcePath.substring(slash + 1);
+ copies.add(sourceBucket + "/" + sourceKey + " -> " + destinationBucket
+ "/" + destinationKey);
+ byte[] data = bucket.equals(sourceBucket) ? objects.get(sourceKey) :
null;
+ if (data == null) {
+ noSuchKey(http, sourceKey);
+ return;
+ }
+ if (bucket.equals(destinationBucket)) {
+ objects.put(destinationKey, data);
+ }
+ respond(http, 200, "application/xml",
+ "<?xml version=\"1.0\" encoding=\"UTF-8\"?><CopyObjectResult>"
+ +
"<LastModified>2026-10-09T00:00:00.000Z</LastModified>"
+ +
"<ETag>\"0123456789abcdef0123456789abcdef\"</ETag></CopyObjectResult>");
+ }
+
private void list(HttpExchange http, Map<String, String> query) throws
IOException {
int maxKeys = query.containsKey("max-keys") ?
Integer.parseInt(query.get("max-keys")) : 1000;
String after = query.get("continuation-token");
diff --git
a/components/camel-aws/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Consumer.java
b/components/camel-aws/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Consumer.java
index 567111b473c0..fdde6c3a05a8 100644
---
a/components/camel-aws/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Consumer.java
+++
b/components/camel-aws/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Consumer.java
@@ -191,11 +191,14 @@ public class Sqs2Consumer extends
ScheduledBatchPollingConsumer {
// update pending number of exchanges
pendingExchanges = total - index - 1;
+ // the receipt handle to delete the message with once the exchange
is done. It is taken now, as the route
+ // may remove the headers or replace the message before it ends
+ final String receiptHandle =
exchange.getIn().getHeader(Sqs2Constants.RECEIPT_HANDLE, String.class);
// add on completion to handle after work when the exchange is done
exchange.getExchangeExtension().addOnCompletion(new
Synchronization() {
@Override
public void onComplete(Exchange exchange) {
- processCommit(exchange);
+ processCommit(exchange, receiptHandle);
}
@Override
@@ -254,14 +257,26 @@ public class Sqs2Consumer extends
ScheduledBatchPollingConsumer {
}
/**
- * Strategy to delete the message after being processed.
+ * Strategy to delete the message after being processed, with the receipt
handle of the
+ * {@link Sqs2Constants#RECEIPT_HANDLE} header.
*
- * @param exchange the exchange
+ * @param exchange the exchange
+ * @deprecated use {@link #processCommit(Exchange, String)},
which the consumer calls instead
*/
+ @Deprecated(since = "4.24.0")
protected void processCommit(Exchange exchange) {
+ processCommit(exchange,
exchange.getIn().getHeader(Sqs2Constants.RECEIPT_HANDLE, String.class));
+ }
+
+ /**
+ * Strategy to delete the message after being processed.
+ *
+ * @param exchange the exchange
+ * @param receiptHandle the receipt handle the message of the exchange was
received with
+ */
+ protected void processCommit(Exchange exchange, String receiptHandle) {
try {
if (shouldDelete(exchange)) {
- String receiptHandle =
exchange.getIn().getHeader(Sqs2Constants.RECEIPT_HANDLE, String.class);
DeleteMessageRequest.Builder deleteRequest
=
DeleteMessageRequest.builder().queueUrl(getQueueUrl()).receiptHandle(receiptHandle);
diff --git
a/components/camel-aws/camel-aws2-sqs/src/test/java/org/apache/camel/component/aws2/sqs/SqsConsumerDeleteMessageTest.java
b/components/camel-aws/camel-aws2-sqs/src/test/java/org/apache/camel/component/aws2/sqs/SqsConsumerDeleteMessageTest.java
new file mode 100644
index 000000000000..dd804478ba08
--- /dev/null
+++
b/components/camel-aws/camel-aws2-sqs/src/test/java/org/apache/camel/component/aws2/sqs/SqsConsumerDeleteMessageTest.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.aws2.sqs;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+
+import org.apache.camel.BindToRegistry;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.model.RouteDefinition;
+import org.apache.camel.support.DefaultMessage;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+import software.amazon.awssdk.services.sqs.model.DeleteMessageRequest;
+import software.amazon.awssdk.services.sqs.model.Message;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * With deleteAfterRead the consumer deletes the message it received once the
exchange is done, with the receipt handle
+ * it received the message with, whatever the route did with the
CamelAwsSqsReceiptHandle header in the meantime.
+ */
+class SqsConsumerDeleteMessageTest extends CamelTestSupport {
+
+ @BindToRegistry("client")
+ private final AmazonSQSClientMock client = new AmazonSQSClientMock();
+
+ @Override
+ protected void doPreSetup() {
+
client.addMessage(Message.builder().messageId("m-1").receiptHandle("handle-1").body("hello").build());
+ }
+
+ @Test
+ void messageIsDeletedWhenTheRouteRemovesTheHeaders() throws Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("hello");
+ startRoute(route -> route.removeHeaders("*"));
+
+ mock.assertIsSatisfied();
+ assertDeleted("handle-1");
+ }
+
+ @Test
+ void messageIsDeletedWhenTheRouteReplacesTheMessage() throws Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("reply");
+ // the new message has none of the headers of the received one
+ startRoute(route -> route.process(exchange -> {
+ DefaultMessage message = new DefaultMessage(exchange);
+ message.setBody("reply");
+ exchange.setMessage(message);
+ }));
+
+ mock.assertIsSatisfied();
+ assertDeleted("handle-1");
+ }
+
+ @Test
+ void messageIsDeletedWithTheReceiptHandleItWasReceivedWith() throws
Exception {
+ // control: the route keeps the headers the consumer set
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceived("hello");
+ mock.expectedHeaderReceived(Sqs2Constants.RECEIPT_HANDLE, "handle-1");
+ startRoute(route -> {
+ });
+
+ mock.assertIsSatisfied();
+ assertDeleted("handle-1");
+ }
+
+ private void startRoute(Consumer<RouteDefinition> steps) throws Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ RouteDefinition route =
from("aws2-sqs://MyQueue?amazonSQSClient=#client&delay=50");
+ steps.accept(route);
+ route.to("mock:result");
+ }
+ });
+ }
+
+ private void assertDeleted(String receiptHandle) {
+ // the delete is sent once the exchange is done, after the mock
endpoint got it
+ await().atMost(10, TimeUnit.SECONDS).until(() ->
!client.getDeleteMessageRequests().isEmpty());
+ assertEquals(List.of(receiptHandle),
+
client.getDeleteMessageRequests().stream().map(DeleteMessageRequest::receiptHandle).toList());
+ }
+}
diff --git
a/components/camel-azure/camel-azure-storage-blob/src/main/java/org/apache/camel/component/azure/storage/blob/BlobConsumer.java
b/components/camel-azure/camel-azure-storage-blob/src/main/java/org/apache/camel/component/azure/storage/blob/BlobConsumer.java
index 26e802fa75b5..295345a3eb79 100644
---
a/components/camel-azure/camel-azure-storage-blob/src/main/java/org/apache/camel/component/azure/storage/blob/BlobConsumer.java
+++
b/components/camel-azure/camel-azure-storage-blob/src/main/java/org/apache/camel/component/azure/storage/blob/BlobConsumer.java
@@ -253,13 +253,15 @@ public class BlobConsumer extends
ScheduledBatchPollingConsumer {
// update pending number of exchanges
pendingExchanges = total - index - 1;
+ // the blob to delete or move once the exchange is done. It is
taken now, as the route may change or remove
+ // the blob name header before it ends (the producer takes the
name of the blob to upload from it)
final String blobName =
exchange.getIn().getHeader(BlobConstants.BLOB_NAME, String.class);
// add on completion to handle after work when the exchange is done
exchange.getExchangeExtension().addOnCompletion(new
Synchronization() {
@Override
public void onComplete(Exchange exchange) {
try {
- processCommit(exchange);
+ processCommit(exchange, blobName);
} finally {
releaseInProgress(blobName);
}
@@ -303,18 +305,30 @@ public class BlobConsumer extends
ScheduledBatchPollingConsumer {
}
}
+ /**
+ * Strategy to move/delete the blob after being processed, the blob named
by the {@link BlobConstants#BLOB_NAME}
+ * header.
+ *
+ * @param exchange the exchange
+ * @deprecated use {@link #processCommit(Exchange, String)},
which the consumer calls instead
+ */
+ @Deprecated(since = "4.24.0")
+ protected void processCommit(Exchange exchange) {
+ processCommit(exchange,
exchange.getIn().getHeader(BlobConstants.BLOB_NAME, String.class));
+ }
+
/**
* Strategy to move/delete the blob after being processed.
*
* @param exchange the exchange
+ * @param blobName the name of the blob the exchange was created from, in
the container of the endpoint
*/
- protected void processCommit(Exchange exchange) {
+ protected void processCommit(Exchange exchange, String blobName) {
try {
final String containerName =
getEndpoint().getConfiguration().getContainerName();
- final String blobName =
exchange.getIn().getHeader(BlobConstants.BLOB_NAME, String.class);
if (blobName == null) {
- LOG.warn("Cannot perform move/delete after read because blob
name is not available in exchange headers");
+ LOG.warn("Cannot perform move/delete after read because the
consumed blob has no name");
return;
}
diff --git
a/components/camel-azure/camel-azure-storage-blob/src/test/java/org/apache/camel/component/azure/storage/blob/BlobConsumerDeleteMoveAfterReadTest.java
b/components/camel-azure/camel-azure-storage-blob/src/test/java/org/apache/camel/component/azure/storage/blob/BlobConsumerDeleteMoveAfterReadTest.java
new file mode 100644
index 000000000000..9cf81003bbe2
--- /dev/null
+++
b/components/camel-azure/camel-azure-storage-blob/src/test/java/org/apache/camel/component/azure/storage/blob/BlobConsumerDeleteMoveAfterReadTest.java
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.azure.storage.blob;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.function.Consumer;
+
+import com.azure.core.http.HttpHeaders;
+import com.azure.core.http.HttpResponse;
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.http.rest.PagedResponseBase;
+import com.azure.core.http.rest.SimpleResponse;
+import com.azure.core.util.Context;
+import com.azure.core.util.polling.LongRunningOperationStatus;
+import com.azure.core.util.polling.PollResponse;
+import com.azure.core.util.polling.SyncPoller;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.models.BlobCopyInfo;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRange;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.azure.storage.blob.options.BlockBlobSimpleUploadOptions;
+import com.azure.storage.blob.specialized.BlobInputStream;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import org.apache.camel.BindToRegistry;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.model.RouteDefinition;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+
+import static org.apache.camel.builder.Builder.constant;
+import static org.apache.camel.builder.Builder.simple;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.nullable;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * With deleteAfterRead or moveAfterRead the consumer deletes or moves the
blob it consumed once the exchange is done,
+ * whatever the route did with the CamelAzureStorageBlobBlobName header in the
meantime: a route may set it to upload a
+ * copy of the blob under another name (the producer takes the name from that
header), or remove the Camel headers
+ * before it calls another system.
+ * <p>
+ * The scheduler of the consumer is not started: the tests call {@code poll()}
themselves. The routes are synchronous,
+ * so the delete or the move is done when {@code poll()} returns.
+ */
+class BlobConsumerDeleteMoveAfterReadTest extends CamelTestSupport {
+
+ private static final String ACCOUNT_URL =
"https://account.blob.core.windows.net/";
+
+ @BindToRegistry("client")
+ private final BlobServiceClient client = mock(BlobServiceClient.class);
+
+ private final Map<String, Set<String>> containers = new
ConcurrentHashMap<>();
+ private final Map<String, BlobContainerClient> containerClients = new
ConcurrentHashMap<>();
+ private final List<String> uploads = new CopyOnWriteArrayList<>();
+ private final List<String> copies = new CopyOnWriteArrayList<>();
+ private final List<String> deletes = new CopyOnWriteArrayList<>();
+
+ @Override
+ protected void doPreSetup() {
+ when(client.getBlobContainerClient(anyString()))
+ .thenAnswer(i ->
containerClients.computeIfAbsent(i.getArgument(0), this::containerClient));
+ blobs("in").add("todo/a.txt");
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedBlobWhenTheRouteRenamesIt() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ BlobConsumer consumer = startRoute("&deleteAfterRead=true",
this::uploadCopy);
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("in/out/todo/a.txt"), uploads);
+ assertEquals(List.of("in/todo/a.txt"), deletes);
+ // the copy the route uploaded is kept
+ assertTrue(blobs("in").contains("out/todo/a.txt"));
+ // and the next poll does not consume the blob again
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedBlobWhenTheRouteRemovesTheHeaders()
throws Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ BlobConsumer consumer = startRoute("&deleteAfterRead=true", route ->
route.removeHeaders("CamelAzureStorageBlob*"));
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("in/todo/a.txt"), deletes);
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedBlobWhenTheRouteRenamesIt() throws
Exception {
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ BlobConsumer consumer =
startRoute("&moveAfterRead=true&destinationContainer=archive",
this::uploadCopy);
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("in/todo/a.txt -> archive/todo/a.txt"), copies);
+ assertEquals(List.of("in/todo/a.txt"), deletes);
+ assertTrue(blobs("in").contains("out/todo/a.txt"));
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void deleteAfterReadDeletesTheConsumedBlob() throws Exception {
+ // control: the route keeps the header the consumer set
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ mock.expectedHeaderReceived(BlobConstants.BLOB_NAME, "todo/a.txt");
+ BlobConsumer consumer = startRoute("&deleteAfterRead=true", route -> {
+ });
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("in/todo/a.txt"), deletes);
+ assertEquals(0, consumer.poll());
+ }
+
+ @Test
+ void moveAfterReadMovesTheConsumedBlob() throws Exception {
+ // control: the destination name is built from the name of the
consumed blob
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedMessageCount(1);
+ BlobConsumer consumer =
startRoute("&moveAfterRead=true&destinationContainer=archive&removePrefixOnMove=true"
+ +
"&destinationBlobPrefix=done/&destinationBlobSuffix=.bak",
+ route -> {
+ });
+
+ assertEquals(1, consumer.poll());
+
+ mock.assertIsSatisfied();
+ assertEquals(List.of("in/todo/a.txt -> archive/done/a.txt.bak"),
copies);
+ assertEquals(List.of("in/todo/a.txt"), deletes);
+ }
+
+ /**
+ * Uploads a copy of the blob to out/ in the same container: the producer
takes the name of the blob from the
+ * header.
+ */
+ private void uploadCopy(RouteDefinition route) {
+ route.setHeader(BlobConstants.BLOB_NAME,
simple("out/${header.CamelAzureStorageBlobBlobName}"))
+ .setBody(constant("copy"))
+
.to("azure-storage-blob://account/in?serviceClient=#client&operation=uploadBlockBlob");
+ }
+
+ private BlobConsumer startRoute(String options, Consumer<RouteDefinition>
steps) throws Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ RouteDefinition route =
from("azure-storage-blob://account/in?serviceClient=#client&prefix=todo/"
+ + "&startScheduler=false" +
options)
+ .routeId("blob");
+ steps.accept(route);
+ route.to("mock:result");
+ }
+ });
+ return (BlobConsumer) context.getRoute("blob").getConsumer();
+ }
+
+ private Set<String> blobs(String container) {
+ return containers.computeIfAbsent(container, k -> new
ConcurrentSkipListSet<>());
+ }
+
+ private BlobContainerClient containerClient(String container) {
+ BlobContainerClient containerClient = mock(BlobContainerClient.class);
+ when(containerClient.listBlobs(nullable(ListBlobsOptions.class),
nullable(Duration.class))).thenAnswer(i -> {
+ ListBlobsOptions options = i.getArgument(0);
+ String prefix = options != null && options.getPrefix() != null ?
options.getPrefix() : "";
+ List<BlobItem> items = blobs(container).stream().filter(name ->
name.startsWith(prefix))
+ .map(name -> new BlobItem().setName(name)).toList();
+ return new PagedIterable<>(() -> new PagedResponseBase<Void,
BlobItem>(null, 200, null, items, null, null));
+ });
+ when(containerClient.getBlobClient(anyString())).thenAnswer(i ->
blobClient(container, i.getArgument(0)));
+ return containerClient;
+ }
+
+ @SuppressWarnings("unchecked")
+ private BlobClient blobClient(String container, String name) {
+ BlobClient blobClient = mock(BlobClient.class);
+ when(blobClient.getBlobUrl()).thenReturn(ACCOUNT_URL + container + "/"
+ name);
+ when(blobClient.openInputStream(nullable(BlobRange.class),
nullable(BlobRequestConditions.class))).thenAnswer(i -> {
+ if (!blobs(container).contains(name)) {
+ throw notFound();
+ }
+ BlobInputStream stream = mock(BlobInputStream.class);
+
when(stream.getProperties()).thenReturn(mock(BlobProperties.class));
+ return stream;
+ });
+ doAnswer(i -> {
+ deletes.add(container + "/" + name);
+ if (!blobs(container).remove(name)) {
+ throw notFound();
+ }
+ return null;
+ }).when(blobClient).delete();
+ when(blobClient.beginCopy(anyString(),
any(Duration.class))).thenAnswer(i -> {
+ String source = i.<String>
getArgument(0).substring(ACCOUNT_URL.length());
+ String sourceContainer = source.substring(0, source.indexOf('/'));
+ String sourceName = source.substring(source.indexOf('/') + 1);
+ copies.add(source + " -> " + container + "/" + name);
+ boolean found = blobs(sourceContainer).contains(sourceName);
+ if (found) {
+ blobs(container).add(name);
+ }
+ SyncPoller<BlobCopyInfo, Void> poller = mock(SyncPoller.class);
+ when(poller.waitForCompletion(any(Duration.class))).thenReturn(new
PollResponse<>(
+ found ? LongRunningOperationStatus.SUCCESSFULLY_COMPLETED
: LongRunningOperationStatus.FAILED, null));
+ return poller;
+ });
+ BlockBlobClient blockBlobClient = mock(BlockBlobClient.class);
+
when(blockBlobClient.uploadWithResponse(any(BlockBlobSimpleUploadOptions.class),
nullable(Duration.class),
+ nullable(Context.class))).thenAnswer(i -> {
+ uploads.add(container + "/" + name);
+ blobs(container).add(name);
+ return new SimpleResponse<>(null, 201, new HttpHeaders(),
null);
+ });
+ when(blobClient.getBlockBlobClient()).thenReturn(blockBlobClient);
+ return blobClient;
+ }
+
+ private static BlobStorageException notFound() {
+ HttpResponse response = mock(HttpResponse.class);
+ when(response.getStatusCode()).thenReturn(404);
+ return new BlobStorageException("The specified blob does not exist.",
response, null);
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index 0b23c1b1d3da..2577b3f8f95b 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -1819,6 +1819,26 @@ section of
xref:components::aws2-kinesis-component.adoc[AWS Kinesis].
`copyBlob` now rejects invalid source account or container names (headers
`CamelAzureStorageBlobSourceBlobAccountName`
and `CamelAzureStorageBlobSourceBlobContainerName`) with an
`IllegalArgumentException`.
+=== camel-aws2-s3, camel-azure-storage-blob, camel-aws2-sqs - the consumer
deletes or moves what it consumed
+
+With `deleteAfterRead` or `moveAfterRead`, the `aws2-s3` and
`azure-storage-blob` consumers now delete or move the
+object they consumed, in the bucket or container of the endpoint. Before, they
took the key and the bucket from the
+`CamelAwsS3Key` and `CamelAwsS3BucketName` headers, and the name of the blob
from the `CamelAzureStorageBlobBlobName`
+header, as the route left them. A route that set these headers, for example to
write a copy of the object under
+another name, deleted or moved that copy, and the consumed object was consumed
again by the next poll. A route that
+removed them deleted nothing.
+
+The `aws2-sqs` consumer now deletes the message with the receipt handle it
received it with. Before, it took the
+`CamelAwsSqsReceiptHandle` header as the route left it, so a route that
removed the headers or replaced the message
+did not delete it, and the message was received again after its visibility
timeout.
+
+A failure of the `aws2-s3` consumer to delete or move an object that the AWS
client raises itself (for example when S3
+cannot be reached) now goes to the exception handler of the consumer, as a
failure that S3 answers does.
+
+The consumers now call `processCommit(Exchange, String)` with the key, the
name of the blob or the receipt handle. A
+subclass of `AWS2S3Consumer`, `BlobConsumer` or `Sqs2Consumer` that overrides
`processCommit(Exchange)` must override
+the new method instead. `processCommit(Exchange)` is deprecated.
+
=== camel-stitch - a response with an empty body, and the default sequence of
a message
A response of Stitch with an empty body now sets the `CamelStitchCode`,
`CamelStitchStatus` and `CamelStitchHeaders`